Momentum and Optimal Stochastic Search

The rate of convergence for gradient descent algorithms, both batch and stochastic, can be improved by including in the weight update a “momentum” term proportional to the previous weight update. Several authors [1, 2] give conditions for convergence of the mean and covariance of the weight vector for momentum LMS with constant learning rate. However stochastic algorithms require that the learning rate decay over time in order to achieve true convergence of the weight (in probability, in mean square, or with probability one). This paper uses the dynamics of weight space probabilities [3, 4] to address stochastic gradient algorithms with learning rate annealing and momentum. This theoretical framework provides a simple, unified treatment of asymptotic convergence rates and asymptotic normality. The results for algorithms without momentum have been previously discussed in the literature. Here we gather those results under a common theoretical structure and extend them to stochastic gradient descent with momentum.